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Defense. Overseasi Dependents Schools followed this pattern and was 
under attack froi lany sides; ^Conseguentlyr the testing pcogxaa was 
suspended in 1971 to provide funds .for an evaluation to, drtetaine the 
lost appropriate type of testinq prcgria to fteet'the needs o^the 
Overseas Dependents Schools., & four step need? assesssent, evai^uation 
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the eleaentafy ind.secondary schools were identified^ Second, the 
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inforaition In each of the areas wjsts ieterained. Third, probleas 
associated with the old testing ptograa and characteristics of the 
Overseas Dei>end#fi;ts Schools that warranted consideration in 
developing ^ testing prograa were identified. Fourth, the infortotion 
was analyzed to deteraine the aost appropriate purposes for testing, 
at each adainistrative level of the systea, and the aost appropriate 
testsi( saapling and' adainistrative procedures for a testing prograa 
to provide the requited inforaation. (iuthor/H?) 
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' School. testing programs >coniponly have been limi^jBd t;o obtaining little 
more than a^ IQ and achievement scores in reading and mathematics for each 
student* Testing of this type is under attack from many sides: sope feel ^ 
the tests are not valid predictors of important skills; minority groups 
argue Klhat the tests are culturally biased; some fear that testing has a 
negative effect on self-esteem; others complain that tests do not measure* 
critical qualities such as honesty or ambition; some argue that testing In 
curricular areas such as art atid science is equally importanty some feel 
that testing is an invasion^of individual pcivacy; and others claim that 
teachers rely \oo rigidly on test results. Many of these complaints Are 
justified; some reflect misunderstandings. 

. Testing in the' U.S. Department of Defense Overseas Dependent Schools 

System (CDS) followed the traditional pattern dhd was subject to all of 

th^ .attacks. The schools of the ODS primarily serve the minor dependent 

children of Department of Defense personnel. Centrally administered from 

, " \ ' 

the Pentagon by the Directorate for Dependents Education, 'the system is 

divided into three areas: Pacific area schools administered by the 

Air Force, Atlantic area schools by 'the Navy, and European schools 

(including schools located in the Middle East and Africa) by the Army. 



/ 



Ulthjan enrollment of almost 175, OOb students. 



it is one of the largest 



,Ameritan school systems. However, ^the rfelatiye isolation from American 
life, the extreme mobility of the iktudents, the frequent ^;urnover of staff, 
difficulties of communications within the system, requirements imposed by 
the laws of host countries, and the vagaries of tntemational relations 

n 

pose problems, that are unique among American schools. 

» 

The problem raised by the Department of Defense in its evaluation of ^ 
the ODS testing program required a decision as to the most important pur- 
poses that it should serve. | Determination of the important purposes of a 
testing program called for a nepds-assessment eva;luation tfiat would collect 
information and process it objectively and would compare present with 
desired practice for each of the purposes identified.^ The steps of the 
needs-assessment evaluation are explained below as they were applied to 
the problem of evaluating the need for test information in the PDS. 
Determining the Purposes the Testing Program Should. Serve 

In an electi6ri, a write-in candidate usually f|as no real chaijce of 
winning. Tbe more complete the \ist of names on the ballot, the more%» 
likely the results will reflect the opiniqft^ of the electorate. Similarly, 
if teachers and^admi^ ^ a^ pCTiplete list of areas in which 

testing might be conducted, the results would reflect the importance of 
various types- of test information. Therefore, the first task in conducting 



the needs. assessment was to dr^iw up a ballot that presented a complete list 
of affective, cognitive, and psychomotor areas in which testing could occur. 



Representative matenfals,, including course syllabi, lists of textbooks 

used, and descriptions of curricular offerings were collected from schools 

throughout the*w)S to develop the required ballot list. 

A major/problem in conducting this type of "election** is establishing 
* / ^ 

a su1;table^level of specificity for the testing purposes. If they are too 
specific, they Jkecppe trivial for guiding a testing, program, and would be 
so namerous that the ballot would be too long to complete^." If the purpose? 

very general they would be perceived to include areas of varying 
Importance and thus would be too ambiguous to rate. Analysis of Materials , 
from the ODS led to development of -a 106 -item ballot for elementary schools 
and a 105 item ballot far secondary schools. Three purposes from each 
ballot ^re presented in Table 1 by way of examples. One ballot for elemen- 
tary and one for secondary school personnel called for them to rate on a 

five-point scale the Importance of having test Information about each area. 

« » ^ * , ^ 

Complete lists of purposes on the two ballots are presented In Churchnan,. 

* ^-^ * " 

Alkin, Hoepfner, and Bradley (1972). 

Insert Table 1 about here * . 
--------- 

Determining the Importahce of Areas In Which Testing Should Occur 

, . The resuT1;s of any election depend to some extent on who is franchtsed 
to vote. In/ collecting needs -assessment information, everyone who will' 
need* the irwormation should be able to express an opinion. The ODS wished 
to determine needs for test information at several levels. Including the 
classrooni teacher-, the school, the overseas area off ices, •and the office 



of the Pentagon. It was necessary, therefore, to obtain information from 
representatives of each of these levels, ^Scliools administered by ealch of 
the thiree services were sample, tr^eating elem^tary and secondary schools 
separately. In this manner, ali, army-administered elementary schools in 
the European area, for^example', were assigned to a single sampling cell, 
and a random sample of the schools in that cell was drawn. Similarly, 
each school in the ODS was assigned to Its respective cell, and a random' 
^ample of .schools in each cell was drawn. ^ > / ^ 

These procedures ensured that schools throughout the OD^ would-be 
sampled, and that each sqhool had an equal opportunity (within each cell) 
of being selected. Other dimensions of the sampling plan were the 
isolation and the size of the school. Isolation of the schools, as 
.measured by distance from other dependent schools, time required to reach 
*the school, number of visits from area-level personnel, hardship ratings, 
artd the like, probabVy affect the attitude of the school personnel. 
School size is important because the number and type of specialized 
personnel and equipment. at a schofol are ||rgely determined by formulas 
that authorize "so^much of this and so much of that" based on school 
enrollment. ' , . 

Ballots were then sent to each school 4n the sample; 107 schools, or 
89% of the sample, returned a total of 846 elementarylind 677 secondary 
ba^llots. Means and ranks, we re computed as measures of the relative 
importance of each of the potential testing areas. 



\ 



5. 



The most noteworthy finding was that there was little agreement among 
the schools ^n'tfie samples. Thirty-five different curriculum areas 
Appeared among the top. five at one or moj^e elementary schools; thirty- 
three different |reas appeared among the top five at one or more secorKlary 
schools. The lack of agreement airtbhg schools in the samples as to the 
most fbportant areas for whic^ each needs test information suggests that 
it is. not justified' to select five or ten or some other arbitrary number 
of areas for testing throughout the entire ODS. Rather^ each school • 
should be free to determine sane pf the areas for which it desires test 
information. 

Each potentra*! area for testing listed^in the ballot was conceived 
aa being part of a larger domain. The curriculum domains that app&red 
at.the top and the bottom of the rankings are presented \r\ Tabl? 2. 



Insert Table 2 about here 



The unexpected *characteri Stic of these results is the clear rejection 

» • 

of the importance of tesrt: information for foreign language skills » which 
contrasts sliarply with the official concern for personal benefits frdm the 
overseas experience; The ballot findings suggest that tests related to 
areas sCrch honesty, creativity, persistence, critical thinking, sports- 
manship, and similar Individual attributes are "^wt perceived as being 
important .for test-ing. V c - 
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•4 . * • . , J ' 

• . ■ / * • - 

". • ; Comparing Present with Desired Purposes for a^Testing Program . • 

• From among the scjraols receiving the ballots., a. smaller sample was 
. selected for visitation- A three-jaart ^strategy for on-sitfe interviews 
■^i was developed thjjt cabled fo^ (lT identification of issues anil concerns 

- affecting testing programs at the first few schodls visite<^ (2) in-depth 
discussion of one or two of these, issues with each of the groups inter- 
viewed at the next few schools, and (3^ verification of findings and 
interpretations at the last few schools visited. The third part of the 
strategy ensured that views expressed by individuals were coranon to orore 
than one school and it allowed the interviewers to obtain reactions to 
» , specific suggestions mad? at othe»<Rchools. . 

The interview schedule was completed by 375 ODS personnel. The first 
set of items in the schedule contained 19 criticisms of tests similar to 
those in the first paragraph of this article'. The resp6ndents indicated 
the extent to which they thought each triticism was a problem in testing 
children in their particular school. The second set contained 20 state- 
ment? (19 for elementary personnel) that -described ^^arious assessment and 
evaluation activities that opcur in school settings. Respondents were to 
• indicate how frequently they had used published tests for each activity. 
. The third set of items rep'^ated thosesof the second set.^but respondents . 
were asked how important it was for them to have that type of informatioh 
in the future. ' ^ . 

f , While widespread xlisagreement was found among schools^s to the cur- 

ricylum areas for whi ch te st information was* needed, .there was strong 
agreemeat among those interviewed, regardless of grade le^l or geographic 
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arfea-, as to the Way tests were used -and should be usedi^ In particular, 
staffs' felt that it takes too long after Ijestfng is completed to get scores 
back, and that the results depend too much ort^ow students feel when they 

take the ttests. Elementary personnel were much less satisfied with once- 

h 

a-year testing than weYe secondary personnel. 

^ Both. groups felt that cultural bias was art importa?it problem with the 
testV Three distinct aspects of this problem were noted, first, there 
tKas the proWem that meiubers of minority groups have With the tests, which 
is the.sairfe as that in the' United States. Second, there was the, problan 
that children of an American soldier-father and a non-American (Korean^ 
Vietnamese, German) mother had with the tests. Third, there were biases 
that stem from differences between civilian and military life. A test 
question that asks children to distinguish the picture 'of d store frtra- 
those of .a hospital and a school is dl^icult for children at military ^ 
bases where buildings are sometimes architectural ly Indistinct. One itan 
of one test asks .the* child to identify the way milk fs delivered, and 
pictures a truck, a plane, and a boat. All three are correct, depending 
upon the base at which the child is stationed! While the number of such 
Items Is small, one or two such items on a subtest can have a significant^ 
.effect upon the child's score. * 

Both the ballots and the interviews suggested that schoo{ staff did 
not view testing as an invasion of privacy", or a cause of exgsssive com- 
petition among students. However, they did view cheating as a major " 
problan of testing, and students with whom this problem was discussed 
viewed competition,- in addition to the repetitfveness of the tests and 
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the lack of infonnation as to r^ults, as a major: cause^ of the 
cheating. . ^ , • / 

The intervj^ees discussed mariy other considerations they felt / 

^important in designing a suQpessful testing program. The relative/ * 
adyantages of teacher vs. specialist administration of tests was- raised, 
with the weight of^opinion in the direction of teacher adrainis'tration. 
The problems of once-a-year testing and rault^le testing were weighe<J 
against student niobility pattern!*. The best hour of the day; length of* 
test, and age af^which separate answer sheets could be introdMced, the(^'4i 
type of training needed by teachers to improve administration of tests 
and factors such as the attitude of the instructor that would influence 

^ the success of the training were discussed. The ways that testing con- 
ditions varied across this worldwide system were e)y?lored in order to 
enabje development of guidelines to improve overall uniformity and thus 
make test results morcr comparable. Problems of coordinating a testing 
program on'^ worldwide bas,is were disfcussed. The interviewers compared 
and interpreted their findings. and- developed a plan to account for 
as many of the problems as possible. » 
Determine Testing Procedures Most Important to Correct ^ 

It was apparent from the interviews that tests were not optimally ' 
used, nor >did teachers want tests to be used for the purposes of grading 
Students, promoting students, ^r accountabiTity of teachers for Student 
learning. Rather, there was a pi ear preference among teacihers for test 
information that l^ould provide diagnostic, plac^ment^ and counseling 



information. It was unfortunate ^t the tests of the extant testing pro- 
gram were not designed for and would ,not be appropriate t^used for those - 
purposes. 

Jhere was little interest a^'^he area and system level in using tests 
to evaluate Individual' teachers, but there was' eoncem l^ith evaluating 
priority and experimental programs. With identifying and cterifying problems 
at the ODS, apd .with reporting mc^e; completely the. accompli sfiments of ^the 
'system to the Congres^ of the United States. ' 

The evidence accujnulated from the ballot and the interviews suggested 
that four major discrepancies should be dealt with in designing the new 
testing program. First, few tea^chers used the information collected • 
because it did n^xprovide the diagnostic information they needed. Second, 
scores from the testing program provided no information about many important 
aspects of the curricuTym such as art, mu-sic, commercial subjects, and the, ^ 
physical and life sdiencesVand/ thus fell far short of measuring the fill 
range -of achievement of the schools. Third, the testing program was ^ 
inadeqaate for makttig decisions abbnt^the effectiveness of priority or 
innovative programs such, as those dealing with minority groups, drugs, or 
career education. Fourth, even in tho^e areas measured by the teisting prp- 
gram^ student mobility patterns in and out of t^e^schools made it impossible 
to interpret the scores as measures of the school S^tti^selves, because tJ^y 
did not identify the source Of the learning that was measured. 



Recommendations . ' ^ • ^ 

. Elimination or reduction of the discrepancies noted above requires 
that testlrtg be conducted for six purposes. ^ 
A. At the school level, testing should be conducted to:* ^ 
T» Diagnose students v^ho appear to be having laming di^fl* 
cultiiBS. .4^his finding is especially crucial at the/ ^ 
elementary level and for reading skills In all^ grades. 
;2\» Plac^nevily arrived students. The* pi acement tests should • 
be short, easily scored, and need yield only ..gross place- * 
meat. data. ^ , ^ 

3. Enable the school staff to evaluate their program at the 
' local level. - 
Bi At the district and ajrea 1^vels$ testing should be conducted to: 
. * Evaluate the implementation, progress, and impact of priority 

and experimental programs. * > 

5. Clarify the na:|ture af problems, such as those associated with 
the cha^att^sti'cs of*troops assigned to a part1cul9||^base« • 

C. At the Department Of Defense level, testing should be conducted to:* 

6. Enable the ) Department of Defense to report to the Congress 
facjtual Inffomiatlon about thelsffectiveness of the ODS. 

In order to mefet these requirements, ^s|j^(?c1f 1c recomnenda^ons were 
made with if^spect to teacher training, testing conditions, student attitudes. 
Implementing the testing program, and the evaluation of prpgram-eva)||>tion* 
systems within the school. ; Tests appropriate for elementary and*§econdary 
schools were Identified, employing the CSE Test Evaluation sarins (Hoepfner, 




et a1, 1^70, 1974) i as 'betng in't|>e topjiridnj^^^^ any 

schooT. A system to ensure ^tbat the' se) f -Evaluation was actountable to 

liigher administration levels' o^-::|he>^s|:em-4*aV 1^^^ ' • . 

The completed report was, neH^ed by^a coimKlttee of Pupil Personnel 

Service directors and selected staff mfembers of the (JDS. . Recoraroendatiohs 

to|al\ow schools mpre responsibility dn developiTig their own testing pro- 
' ^ • \ * • 

grams, and to implement a sampling plan to maetNQdS information needs were 

adapted to administrative requi remerjfs a^nd a new pragraiftjof testing mo.re 

accurately tailored to the needs of the dependents feducation system ha^ 

been' implemented'.' . , • *- ' . - 
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X We would like to thalik .Dr. Thomas Orysdale, Deputy Director of' 
^ ^ Dependents Education, for reviewing this article and adding the note on the 



i^way the evaluation report was used by CDS. 
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) „■ • . 

Saniple Purposes* of the Testing Prosram from the ODS Ballots"*^^ 

^ - . . Elemejntary Lev«V Testjng Program ' 

GROUP ACTIVITY - SPORTSMANSHIP . ' • ' \ / 

Is a. good winner and. a good loser. ^Can be a leader or a follower. 
Obeys the rules of the game. Feels very involved in the sport 
tealn spirit,* • ■ ' • 

INfERENC^ MAKING FROM R^ADHiG SZLECTIONs' * . . ^ 

• • Correctly interprets what is read. " Recogrtizes from the irtateriaVrea'd 
what kind of characters are being ta;lked about. Can tell that the 
characteirs in a story are sad, happy; trustworthy » or not to be 
trusted,^ fete-, ^an»tell why characters act a^ they do. ' - 

^."v ,lp" '• ■ \ ' " 

SYSTEf^TIC REASONING 

- Produces and solves complex problems and evaluates their solutions. 
Considers all ttie elements in situation .and arrives at solutions 
. -through ^deductive reasoning. * , • 



Secondary Level Test i rig Program 




AtriTUDE TQWARD THE WORLD, OF WORK ^ \ # 

Is interested in understanding the world 'of work,* both in it,s. larger' 
scope and as rt relates to own chosen vocation. • • * ' 

WRITINfi / . ' ; , " . ' - . : ' 

Expresses self c>«fcrly in writing, using adequate vocabulary and' gram- ' 
matically correct sentences. Know3,the various types of writing ' ' . 
• (narrative, descrtptive, argumentative, ^and persuasive), and crganizes ^ 
own writing. Knows and uses the.ru.les governing various sp^ial written • 
forms such as. letters, ^ppTicatfoijf , prders, and Scientific reports. .V? 

VISUAL'. ARTS x . , 

Has a "ba si cinders tan Ai rig of the nature and scope of thf visual -^rt%. , 
Is familiar«wtth the various jTiedia,;techniques'» and styles of the t 
visual .arts. Is familiar with historical and contemporary works, of 
art in this and other cultures. Is able to analyze and criticize works 
of art. . - . 



\ Table 2 



' Most and Least Important Domains for the ODS Testing Program 



Elementary Level .Curriculum Domjiins 



Mo^t Ifqponant ^ Uast Imponant 

, ^ - ■ * — ■■ ■ — — — 

Affective . • ^ - Arts and 'Craf£s 

Language Construction Foreign Language < 

Arithmetic Concepts • . ' Music ' • 



» ^- Religion 



Secondary Level Curriculum Domains 



y Most Important ^ • Least Important 

— ■ — — — ■ . • — ^ 



Meptal Health ^ ^ I Mathematical Skills 

Intellectual^ Functiohfng \ Knowledge, of Arts - 

Vocational Competence , . ' Understanding of -Jiature 
. • ' Understanding of 'Technology 
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